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10 August 1977 

The ORD Career Service Panel has asked MAG to look into 
the question of program evaluation. During the past several 
months we have looked into the matter and read many papers 
and a book on the matter. We have not been able to produce 
a formula for program evaluation. We have explored the 
question of why it seems to be such a difficult problem. 

One of the observations we had is that most R&D evaluation 
is done by an analysis of return on investment (ROI). In 
the intelligence community no value has been assigned to” 
ineelidecnes reports or international estimates. That in 
turn makes it difficult to use any ROI method. 

One key reason we have come up with is that most of 
our R&D work is removed aes or more from the final consumer. 
Many of our products go from us to an engineering organization 
and we have a smaller control over the final customer use of 
that product. A simplified process is exhibited in Figure 
1. We find that we have to justify a project before the 
process begins and then despite whatever point ORD's role 
ends we have to explain or evaluate in terms of the ultimate 
product produced whether by us or another organization. 
This is a critical point especially when we realize when one 
of the key criteria in determining project success is whether 
the project results in ultimate adoption of the product for 


consumer use. 
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How do we increase the probability of ultimate consumer 
use? 

How do we show a customer the possibility of a product? 

How do we convince the customer that the technology 
proposed makes sense? | 

How do we find what the customer's problems are? 

What his needs are? | 
CONTACT 

Contact has implications. It means: 

A. Sensitivity to major consumer problems-they 
will expect us to grasp their nraerieies; to know what 
their most difficult jobs are. 

B. Conscienceness of multi-disciplinary application 
(matching) of technological possiblity to their problems-- 
they will expect us to be above internal ORD parochialism. 

Cc. Delivery of the most cost effective solution 
(product) to apply against the problem: 

(1) It may not be the best technical solution. 
(i) Better technology may take more time 
(ii) Better technology may cost more 
(2) It may mean scrapping the R&D project 
(i) To pursue a different technological 
path to the solution 
. (ii) To wait concerning this problem to 
attack another problem because: 
(a) The cost may be high 


(b) The technology may not be here yet 


Approved For Release 2007/10/23 : CIA-RDP91-00452R000100060030-9 


ee 


e f ; Approved For Release 2007/10/23 : CIA-RDP91-00452R000100060030-9 
ow’ ~~“ 


Contact has advantages. It means: 
A. <A better chance of understanding the consumers 
problem. | 
B. A better chance of applying the right technology. 
c. <A better chance of maintaining the customer 
interest all the way. . 
D. A better chance of selling the product. . oH En im 
SUGGESTIONS AND RECOMMENDATIONS | 
1. Division staff chiefs should undertake internal 
programs to assure systematic and increased contact. This 
should be put into their respective LOI's by D/ORD and they 
in turn should put such a contact requirement into the LOI's 
of individual project officers. 
2. The Management Staff should undertake a program of 
basic project officer education. A requirement of each 
project officer should be to take the course "Project 
Officer in the Contract Cycle". Perhaps ORD could work out 
an arrangement with OTR to offer this course on 4 part-time 
basis in Ames Building. Then MS should make other information 
available which will explain the process, the paper requirements 
such as the notebook, the contract inspection reports, etc. 
3. PAS should be a clearing house for interdisciplinary © 
projects or projects requiring a different discipline than 
the main contact division. 
4. We strongly recommend that the Director of Research 
and Development implement a plan which will permit divisions 
to cut projects off in midstream without penalty, i.e., the 


division should have the capability to apply the remaining 
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fund to another technological approach of the same problem 
or to the next priority unfunded problem on that division's 
list. We feel this is one the key reasons why certain 
_projects are not cut off earlier. This would require that 
the Contract Staff would have to write the type of contract 


which would enable such cutoffs. 
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